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ABSTRACT 



The Michigan School Readiness Program (MSRP) is the state's 
early childhood education program for 4-year-olds at risk of school failure. 
This report details previous findings from the longitudinal study of two 
cohorts of MSRP children from kindergarten through the primary grades, 
current findings from the 1998-99 school year, program quality and risk 
factor data, and the grantee evaluation support project. The program includes 
a child development component providing age-appropriate activities to promote 
intellectual and social growth, and a family support and guidance component. 
Year 3 of the evaluation focused on testing for Cohort 1 MSRP program effects 
over and above effects of key background variables. Findings indicated that 
after controlling for site differences, participants' characteristics, and 
socioeconomic status, MSRP students remained significantly higher in overall 
development in kindergarten and received higher ratings on some aspects of 
school readiness, compared to non-MSRP students. The Cohort 1 MSRP students 
had a lower grade retention rate than the comparison group by the end of 
Grade 2. Site variability in program effects suggested a need for further 
development in curriculum, instructional staff, and philosophy. Also, MSRP 
program quality was generally high, although lower quality was found related 
to organizational and instructional issues and lack of outdoor play space. 

The rate and prevalence of child risk factors have been quite stable. A 
Grantee Evaluation Support Project (GESP) provides training workshops, custom 
support activities, and print materials to improve local programs. Findings 
regarding the GESP revealed the need for quantitative data to substantiate 
progress. (Data tables are appended.) (KB) 
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Executive Summary 

Every day, more than 20,000 four-year-old children attend the Michigan School 
Readiness Program (MSRP) in 470 public school districts and 66 other agencies across the state. 
The MSRP is one of the 31 state-initiated and -funded preschool programs in the U.S. that are 
designed to help poor or other children at risk of school failure start school ready to learn. Is the 
Michigan program achieving this goal? In order to answer this question, the Michigan State 
Board of Education awarded a grant to the High/Scope Educational Research Foundation 
beginning in 1995 to design and conduct an evaluation to assess the implementation and 
effectiveness of the Michigan School Readiness Program. The program had not been evaluated 
since its inception in 1985. 

For the last three years, the MSRP Evaluation’s Longitudinal Study has followed two 
cohorts of children from kindergarten through the primary grades. The two cohorts were selected 
in seven sites from across the state. Each cohort consists of a group of children who had 
participated in the MSRP and a comparison group who are like the MSRP children in age and 
socioeconomic background, but had not attended the MSRP. Every year since kindergarten, data 
have been collected on the study participants’ developmental outcomes and school readiness; 
attendance, grade level and special services from school records have been reviewed; and 

• . ■ t 

parents have been interviewed concerning their involvement in their child’s education. 

The results of the first two years’ assessment were presented in Early Returns and the 
Semiannual Progress Report of March, 1999. The reports presented evidence of these MSRP 
effects: 
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• For Cohort 1 participants (who were in kindergarten in the first study year and Grade 1 in 
the second study year), in their kindergarten year, outside observers rated the MSRP 
group significantly more advanced in initiative, social relations, creative representation, 
music and movement, language and literacy and overall development than its 
comparison group. Teachers also rated the MSRP children significantly higher overall on 
the School Readiness Rating Scale than the comparison children. In Grade 1, teachers 
rated the MSRP group significantly higher than its comparison group on several items of 
the School Readiness Rating Scale. 

• For Cohort 2 participants (who were in kindergarten in the second study year), teachers 
rated the students who had attended the MSRP as significantly more developmentally 
ready for learning overall and in several major areas than the students who had not 
attended the program. 

Because not enough of the participants’ background information had been collected, such 
factors were not taken into consideration in the assessment of the program outcomes noted 
above. Having supplemented part of the missing data in the participants’ background, the third- 
year study focused on testing for the Cohort 1 MSRP program effects over and above the effects 
of key background variables. After statistically controlling for the study site differences and the 
participants’ characteristics (e.g., age, gender) and socioeconomic status (e.g., parental level of 
education, father at home or not, household income, household size), the new comprehensive 
analysis confirmed the following previously found program effects at kindergarten and found 
new program effects through the second grade as follows: 



• The students who had participated in the MSRP remained significantly higher in overall 
development on the Child Observation Record and 5 out of the 6 subscale scores in 
kindergarten, compared to the students who had not participated in the program. 

• The students who had participated in the MSRP had significantly higher ratings on some 
items (e.g., ready to learn, retaining learning, good attendance, interest in school work) 
of the School Readiness Rating Scale from kindergarten through Grade 2 than the 
students who had not participated in the program. 
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Preliminary analysis in this year also found: 

• The Cohort 1 MSRP group had a significantly lower grade retention rate than the 

comparison group by the end of Grade 2 (8% versus 15%). 

These results indicate an overall desirable MSRP effect on child development and 
readiness for school participation. 

The comprehensive analysis also revealed significant variation in program effects across 
the study sites: MSRP effects were found in some sites, but not in others. While examining 
many possible methodological reasons to explain the program-by-site effects, the significant 
differences in the program quality across the six study sites suggest a need for further 
development of the MSRP at some sites, especially in curriculum, instructional staff, and 
philosophy. Besides, an analysis of risk-factor effects on the MSRP children’s developmental 
outcomes indicated that MSRP children who had more risk factors had lower COR scores, which 
suggests the importance of taking the risk factors into account in the future study design. 

MSRP program quality, reported for each classroom each year using the Program Quality 
Assessment instrument is quite high for programs overall - between 4 and 5 on the 5-point PQA 
scale. Low-scoring PQA items (3 or below) cluster around organizational issues (e.g. 
professional affiliation and continuity of trainer), lack of outdoor play space, and instructional 
issues such as use of small-group instruction and use of planning and recall with children’s self- 
selected activities. 

Risk factors - family background and environmental conditions that put children at risk 
of school failure - are tabulated and reported by MSRP program staff each year for entering 
children. Low Family Income and Single Parent are the most frequently cited of the 25 risk 
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factors. On average, slightly more than four risk factors are reported for each MSRP child. Both 
the prevalence and rate of incidence of risk factors have been quite stable over the two program 
years 1996-97 and 1997-98. 

In 1998, with funding from the W. K. Kellogg Foundation and the Michigan Department 
of Education, High/Scope launched the Grantee Evaluation Support Project (GESP) to help local 
MSRP projects use evaluation results to support the quality of their programs. Now in its second 
year, the three-year GESP reaches out to local programs with training workshops, custom support 
activities, and print materials designed to enhance the capacity of local programs to carry out 
their local program evaluations. 

The 1999-2000 GESP training program includes seven one- and two-day workshops (up 
from three the first year) on topics ranging from preschool program evaluation design to 
individual evaluation consultations. To date, GESP training activities have been conducted in 50 
of the 57 state ISDs, whose cooperation was sought in setting up the project. In its first year and 
a half, the GESP has provided evaluation training to over 1680 early childhood staff in over 70 
workshops and conferences statewide. This means that GESP training has reached 
approximately 50% of the MSRP workforce. 

In addition to participants’ workshop evaluations, the impact of the GESP is monitored in 
the Narrative Summary Reports submitted each year by MSRP programs to their state project 
officers. A content analysis of early reports (prior to the GESP) indicates that less than a third of 
MSRP program have been using a child outcome measure capable of demonstrating reliable 
gains for children involved in the program. In these same reports, barely 10% of MSRP program 
used this or other quantitative data to substantiate progress toward the goals of their local 



programs. These findings demonstrate the broad need for enhancing local evaluation capacity. 
There is anecdotal evidence that GESP training is having an impact on the quality of local 
program evaluations and we plan to continue the analysis of Narrative Summary Reports for 
evidence of this impact as more of the MSRP community has the opportunity to implement the 
GESP training. 




v 



13 



Part 1 -- State Longitudinal Study 



The High/Scope Educational Research Foundation was awarded a grant by the Michigan 
State Board of Education to design and conduct an evaluation of the implementation and 
effectiveness of the Michigan School Readiness Program. The effectiveness and quality of the 
program had not been evaluated until the current evaluation study began in 1996. By September 
1999, the State Longitudinal Study had completed its third year. This annual report begins with 
a brief review of the previous findings of the study, then presents a summary of its current status 
and findings and the data collection plan for the next two years. 

Overview 

In this section we provide a general description of the Michigan School Readiness 
Program, a brief review of the longitudinal evaluation research questions and design, and then a 
summary of the findings for the first two years. 

What is the Michigan School Readiness Program? 

The Michigan School Readiness Program (MSRP) is a preschool program funded and 
sponsored by the State of Michigan. It provides nine months of educational experiences to four- 
year-olds identified as being at risk of school failure. Children in the MSRP receive a child 
developmental preschool program that provides age-appropriate activities in order to promote 
their intellectual and social growth and school readiness. Children’s families receive parenting 
support, guidance, and referrals to community service agencies as needed. The MSRP initiative 
began as a small pilot project in 1985 and has grown steadily in the last decade. During the 
1996-1997 school year, the programs operated in 460 of the state’s 560 school districts and 66 
other agencies throughout the state, serving 21,077 children and their families. By the 1999- 
2000 school year, the program had served a total of 212,000 children since its inception. 

Research Questions and Research Design 

Research Questions 

The research questions of the MSRP State Longitudinal Study flow from two major 
intended effects of the program: 

• Does the program contribute to children ’s development and readiness for school 
participation? This question focuses on an evaluation of the program effects on the 
children who participate in the program. 

• Does the program help parents contribute to children ’s development and readiness for 
school participation? This question focuses on an evaluation of the program effects on 
the parents of the participating children. 
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Study Participants 



Study cohorts and sites. The study has two cohorts. Cohort 1 consists of two groups of 
children — a group who attended MSRP classes in 1995-1996 and a comparison group of same- 
age children who did not participate in the MSRP. Cohort 2 consists of three groups of 
children — a group who participated in the 1996-1997 MSRP, a group who participated in Head 
Start, and a group who did not participate in a preschool program. The study sites were selected 
from across Michigan (see the map on the next page). 

• Cohort 1 children were identified from 6 study sites: Detroit Public Schools in the 
southeast of the state, COOR Intermediate School District in the north, Muskegon Public 
Schools in the west, Kalamazoo Public Schools and the Wyoming/Godwin Heights/ 
Godfrey Lee/Kelloggsville School District Consortium in the southwest, and the 
Economic Opportunity Committee of St. Clair County in Port Huron in the east of the 
state. 

• Cohort 2 children were identified from 5 study sites: Detroit Public Schools, the COOR 
Intermediate School District, Muskegon Public Schools, the Wyoming/Godwin Heights/ 
Godfrey Lee/Kelloggsville School District consortium, and the Downriver Guidance 
Clinic. (Downriver joined the study, while Kalamazoo and Port Huron dropped out for 
Cohort 2.) 

Identification and selection of study participants. For both Cohorts 1 and 2, potential study 
participants were first identified as they entered kindergarten. The study comparison children 
from each cohort were identified based on three criteria: 

• They entered kindergarten in the same year as the MSRP children. 

• They did not have a preschool program experience, although a few 
had been in child care. 

• They came from families whose income was low enough to have 
qualified them for the MSRP, based on parents’ self-reported 
income on the Child and Family Background Questionnaire. 

The primary bases for selection of the children into the study were: (a) the data collected 
from parents (through a form sent home and subsequent interviews with parents on the 
telephone); (b) school records; (c) kindergarten teachers’ knowledge of prekindergarten 
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experience; and (d) risk factor information. In addition, for most sites, federal Free and Reduced 
Lunch Program eligibility information was obtained to verify children’s eligibility for the study. 
From these available data, children were identified as being in one of the study groups: MSRP or 
comparison for Cohort 1 , and MSRP, comparison, or Head Start for Cohort 2. 

Sample size. Tables 1 and 2 present the numbers of participants who were identified as 
Cohort 1 and Cohort 2, by group and study site. 



Table 1.1. Cohort 1 Identified Sample Size by Group and Site 





Number of Participants 


Site 


MSRP 


Comparison 


Total 


COOR Intermediate School District 


50 


49 


99 


Detroit Public Schools 


57 


55 


112 


Kalamazoo Public Schools 


53 


38 


91 


Muskegon Public Schools 


75 


37 


112 


W^oming/Godwin/Godfrey/Kelloggsville Public 


51 


46 


97 


Economic Opportunity Committee of St. Clair 
County 


65 


47 


112 


Total 


351 


272 


623 



Table 1.2. Cohort 2 Identified Sample Size by Group and Site 



Site 




Number of Participants 




MSRP 


Comparison 


Head 

Start 


Total 


COOR Intermediate School District 


49 


25 


42 


116 


Detroit Public Schools 


51 


65 


55 


171 


Muskegon Public Schools 


71 


68 


80 


219 


Wyoming/Godwin/Godfrey/Kelloggsville 


37 


41 


28 


106 


Downriver Guidance Clinic 


6 


7 


4 


17 


Total 


214 


206 


209 


629 



Characteristics of study participants. The family background of the participants and 
incidences of risk factors for the MSRP participants were examined for each cohort. For those 
study participants with data collected during their'kindergarten years, the analyses of the parental 
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level of education, proportions of households headed by a single parent, and household income 
indicated no significant differences between the MSRP and the comparison groups for either 
cohort (see Table 3 and Table 4). However, two significant differences in the background 
characteristics of the study groups were found. In Cohort 1, the MSRP group had a lower 
number of household members than the comparison group. In Cohort 2, the only difference was 
between the MSRP and Head Start groups, with Head Start families having a lower average 
income. 



Table 1.3. Cohort 1 Family Demographics by Study Group 



Variable 


Group 


n 


Average / 
Percentage 


Statistical 
Significance 
of Difference 


Mother’s highest year of 


MSRP 


246 


12.04 




education 


Comparison 


186 


11.94 




Father’s highest year of 


MSRP 


154 


12.19 




education 


Comparison 


126 


11.79 




Number of people per 


MSRP 


251 


4.46 


pc. 05 


household 


Comparison 


194 


4.78 




Household annual 


MSRP 


251 


$17,587 




income 


Comparison 


194 


$17,195 




% of father in home 


MSRP 


251 


61% 






Comparison 


194 


60% 
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Table 1.4. Cohort 2 Family Demographics by Study Group 



Variable 


Group 


n 


Average / 
Percentage 


Statistical 
Significance 
of Difference 




MSRP 


157 


12.12 




Mother’s highest year of 
education 


Comparison 


115 


11.90 






Head Start 


132 


12.10 






MSRP 


113 


12.13 




Father’s highest year of 
education 


Comparison 


68 


11.60 






Head Start 


84 


11.78 






MSRP 


162 


4.59 




Number of people per 
household 


Comparison 


115 


4.57 






Head Start 


139 


4.43 






MSRP 


137 


$16,898 




Household annual 


Comparison 


99 


$15,874 


MSRP > Head Start 


income 


p<. 05 




Head Start 


119 


$14,778 






MSRP 


159 


69% 




% of father in home 


Comparison 


116 


57% 






Head Start 


139 


59% 





Figure 1 compares the incidence of 24 risk factors for the MSRP groups of Cohorts 1 and 
2 versus MSRP children state-wide in 1996. The patterns are similar, with a few exceptions: for 
both study cohorts, the MSRP groups had a 5 to 10 percentage -point higher incidence than the 
state-wide MSRP children of the following risk factors: teenage parents, family density, low 
birth weight, and child abuse and neglect ; and 5 to 10 percentage points lower of language 
deficiency and substance abuse. In addition, compared with state-wide MSRP children, Cohort 1 
MSRP children had a lower incidence of unemployed parents and Cohort 2 MSRP children had a 
lower percentage of rural housing. All of these variations are sufficiently small and few among 
24 factors to indicate that the two study cohorts were representative of the state-wide MSRP 
population. 
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Figure 1: Cohort 1 & 2 MSRP vs. All MSRP for Risk Factors 
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Measures and Data Collection 



For each cohort, the following data were collected in order to answer the research 
questions noted above. 

Child Outcomes: 

• The High/Scope Child Observation Record (COR), which measures child development in 
six areas: initiative , social relations, creative representation, music and movement, 
language and literacy, and logic and mathematics. The data were collected in 
kindergarten by trained outside observers. 

• The School Readiness Rating Scale (SRRS), formerly the Child Development Rating 
Scale, which measures child development in school readiness for learning every school 
year from kindergarten onward. Eleven items were rated in kindergarten, and a few more 
items have been added to assess the expanding learning and developmental domains as 
children progress to the higher grades. The ratings were completed by the teachers of the 
participating children. 

• School Records Review, which provides information about the special services the 
children were referred to and used and information about school attendance. It also 
provides risk factor information for MSRP children. The data were collected every year, 
usually by school district staff. 

Participants’ Family Background: 

• The Child and Family Background Questionnaire, which provides child socioeconomic 
information and preschool program participation information collected from parents at 
kindergarten. 

Parent Outcomes: 

• The Parent Interview, which provides information about parent involvement in child- 
related activities both in school and at home, as well as parents’ expectations for their 
child’s schooling. The interview now also includes family background information to 
follow some of missing data from the initial data collection. The interview was 
conducted, usually over the phone, by High/Scope staff in the summer of each year from 
kindergarten onward. 
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Program Quality: 

• The High/Scope Program Quality Assessment, which assesses MSRP classrooms in nine 
areas: philosophy, population access, curriculum, learning environment, advisory 
council, parent involvement, funding, administration and supervision, and instructional 
staff. The assessment was conducted by trained data collectors when the participating 
children were in the preschool programs. 

Data Analyses 

Descriptive and comparative analyses were the major statistical approaches employed in 
the first two years of the evaluation. Children’s outcomes were compared each year to test for 
statistically significant differences between the MSRP and comparison groups in child 
development and school readiness. Because the number of study participants for which we have 
family background data was only about two-thirds of the identified samples, neither background 
characteristics nor socioeconomic status were taken into account in the first two years of data 
analysis. 

Previous Findings 

As shown in Table 5, the kindergarten and first-grade findings for Cohort 1 and the 
kindergarten findings for Cohort 2 were previously reported in Early Returns and the Semiannual 
Progress Reports of March, 1999. The following is a brief summary of the major findings. Not 
many findings regarding parent outcomes have ever been reported because of the limited amount 
of Parent Interview data collected each year. 

High Program Quality was Found for Both Cohorts 

Forty-nine MSRP classrooms were assessed by trained observers for Cohort 1, and 32 
were assessed for Cohort 2. The average total scores on the Program Quality Assessment (PQA) 
for both cohorts ranged in levels between good and excellent. Approximately 50 percent of the 
assessed programs in both cohorts had high-level PQA total scores (average ratings of 4.50 to 
5.00), and no program had a low-level score (1.00 to 2.99). 

Analyses without Background Adjustment Revealed Significant Child Outcomes for the 
MSRP Groups 

Cohort 1. In kindergarten, observers rated MSRP children significantly higher than the 
comparison children on COR total scores and on 5 out of the 6 subscale scores: initiative, social 
relations, creative representation, music and movement, language and literacy. In addition, 
MSRP children received significantly higher ratings from their teachers on initiative, imaginative 
and creative use of materials, and retaining learning. Nearly significant trends (p between .05 
and . 1 0) were found for completing assignments and ready to learn and participate. The 
kindergarten School Readiness Rating Scale total score was significantly higher for the MSRP 
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group than the comparison group. At first grade, MSRP children received significantly higher 
ratings on good attendance, retaining learning, and literacy skills. 

Cohort 2. In kindergarten, teachers rated students who had attended the MSRP as 
significantly more developmental^ ready for learning in several major areas: having good 
attendance, taking responsibility for their own errors or problems, retaining learning, and being 
ready to learn and participate in school. In addition, two nearly significant trends in favor of 
MSRP children were found for showing initiative and completing assignments. The kindergarten 
SRRS total score was significantly higher for Cohort 2 MSRP children than for the comparison 
group. On the other hand, no difference in COR scores was detected between the MSRP and 
comparison groups for Cohort 2. This lack of difference is inconclusive, however, because 37% 
of the COR data were missing and the reliability of the COR for Cohort 2 was very low. 



Table 1.5. Previous Findings (without Background Adjustment) 



Type of 
Assessment 


Cohort 




Cohort 1 


Cohort 2 


Program 

Quality 

Assessment 


High-level: 50%, low level: 0% 


High level: 44%, low level: 0% 


Child 

Outcomes 


COR total & 5 subscale scores in favor of MSRP 


No significant differences on 
COR 




SRRS total & 5 item scores in favor of MSRP 


SRRS total & 6 item scores in 




(Kg.) 

SRRS 3 item scores in favor of MSRP (Gr.l) 


favor of MSRP (Kg.) 


Parent 

Outcomes 


Pending 


Pending 



In summary, the child outcomes strongly favored the MSRP. However, these results must 
receive further analyses to take into account the participant background information. A 
comprehensive test for MSRP program effects became the major task of 1998-1999 year’s study, 
as described in the next section. 



Current Status of the Study 

This section provides a summary description of the Cohort 1 and 2 data collection and 
analyses from 1998 to the present. In the 1998-1999 school year, Cohort 1 study participants 
were in second grade, and Cohort 2 participants were in first grade. In 1999-2000, Cohort 1 is in 
third grade and Cohort 2 is in second grade. 
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Data Collection for the 1998-1999 School Year 
Status of Study Participants 

For each study cohort, the study design projected data collection for three types of 
assessment beyond the kindergarten assessment up to the fourth grade, as feasible. The 
assessments were the School Readiness Rating Scale (completed by the teachers), the Parent 
Interview (conducted by telephone interviewers), and the School Records Review (collected by 
local school staff). Table 6 presents the numbers completed each year by type of data for both 
cohorts. 



Table 1.6. Number of Study Participants by Type of Data and Grade 



Type of Data / Grade 






Cohort 1 

(n=623) 








Cohort 2 

(n=629) 




Kg. 


Gr.l 


Gr.2 


Kg. 


Gr.l 


n 


% 


n 


% 


n 


% 


n 


% 


n 


% 


Family background information 


448 


72 


— 




(+69) 




425 


68 


(+17) 




Child Observation Record 


464 


74 


— 




— 




399 


63 


— 




School Readiness Rating Scale 


499 


80 


445 


71 


360 


58 


349 


55 


322 


51 


School records review 


513 


82 


317 


51 


273 


44 


438 


70 


207 


33 


Parent Interview 


179 


29 


265 


43 


278 


45 


255 


41 


165 


26 



Note. The percentages are calculated on the basis of the total sample size (623 for Cohort 1 , and 629 for Cohort 
2). Family Background and COR data were collected only in kindergarten. Numbers in parentheses for the 
Family background information are the key socioeconomic data derived from the Parent Interview data 
collection this year. The Family background information is now completed for 517 study participants (82%) for 
Cohort 1 and 442 study participants for Cohort 2 (70%). 

We can see from Table 6 that the data received were not complete even in the first year of 
the study, and the response rate decreased with each year on the whole. Attrition is likely in any 
longitudinal study. In our study, the attrition was found to be due to three major reasons: (1) 
family relocated with no forwarding information; (2) family could not be contacted; or (3) 
family, teacher or school declined to provide information. These problems have been worsening 
with the progression of the study despite our many efforts to locate each participant and to collect 
each possible type of data. 

Achieving a Higher Rate of Study Participation 

To compensate for the study attrition and still keep a feasible workload, we used several 
data collection strategies in the 1998-1999 data collection. First, we gave priority in Parent 
Interview data collection to those study participants whose Child Observation Records (COR) or 
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School Readiness Rating Scale (SRRS) data from kindergarten had been complete, but whose 
family background information had not. Second, we gave priority to Cohort 1 over Cohort 2 
study participants, because their data have been much more complete. Third, we added family 
background items to the Parent Interview, thus allowing us to establish the socioeconomic status 
of a higher proportion of the sample in the assessment of child outcomes when comparing the 
MSRP group to the no-program group. 

Although the Parent Interview has always been the most difficult data to collect, we have 
completed 278 parent interviews for Cohort 1 this school year, the highest number of this kind 
ever completed in any of the study years. In particular, we collected family background data 
from 86 parents for whom we previously had no information. In order to achieve a complete data 
set for an about 75 percent of the defined sample for Cohort 1 by the fourth grade, we have 
decided to continue the 1998-1999 year’s Parent Interview data collection during the 1999-2000 
school year. This strategy acknowledges the three important roles that Parent Interview data 
collection is now playing: (1) finding the “lost” participants; (2) completing the family 
background information; and (3) obtaining parent involvement and parent expectation 
information. In addition, we are working with the school systems to achieve continuity of 
cooperation in order to facilitate a more complete SRRS and school records data collection for 
the subsequent school years, which will also help us to collect the missing data from school 
records for the previous school years. 

Estimating MSRP Effects Using a Comprehensive Analysis Approach 

The major purpose of the MSRP evaluation is to assess the extent to which the program 
contributes to children's development and readiness for school participation. In our previous two 
years of analysis, Cohort 1 MSRP students were found to achieve significantly higher scores than 
the comparison group students for both the COR and SRRS assessments; and the Cohort 2 
MSRP group achieved significantly higher SRRS scores than its comparison group. However, 
particularly because the study participants had not been randomly assigned to study groups and 
because there may be other potential bias in the data collection and attrition, it is still necessary to 
estimate the influence of other factors (e.g., family background, participant and site 
characteristics) in the analyses of potential MSRP effects. 

In the previous presentations of the family background information for the MSRP and 
comparison groups in the Early Returns (September 30, 1997 ) and the Semiannual Progress 
Report (March, 1999), we had not found significant group differences in major family 
background variables. Nevertheless, we could not yet conclude that the differences we found in 
favor of the MSRP group were attributable to the MSRP for the following reasons: ( 1 ) The 
reported family background information was not complete (30% was not collected); (2) Students 
with complete family background information did not always have complete outcome assessment 
data; and (3) Additional, related factors, such as site differences or participant characteristics that 
may influence outcomes had not yet been taken into account in the analyses. Therefore, in the 
analyses of MSRP effects in 1998-1999, we focused on taking into account the participants' 




1 - 12 



25 



socioeconomic situation along with other factors that might have influenced the participants’ 
school readiness and child development outcomes. Table 7 presents the key variables selected 
for this comprehensive analysis of MSRP effects. 

Key Covariates Selected 

The key covariates we selected for the families' socioeconomic status were mother’s 
highest years of education, household annual income, father living at home or not, and number 
of people in the household. In numerous research studies, these variables have proved to be 
strong predictors of children’s development. We decided not to include father’s education in the 
analysis because of the low response rate (about one-third of those families with family 



Table 1.7. Key Variables Selected for Comprehensive Analysis of MSRP Effects (1998- 
1999) 



Covariates 


Outcome Variables 


Program status (MSRP vs. Comparison) 

Site (study sites where data were collected) 

Gender (male vs. female) 

Age (in months) 

Mother’s highest years of education 
Father at home (father living at home or not) 
Household size (number of people in the household) 
Household annual income (in dollars) 


Child Observation Record scores (Kg.) 

School Readiness Rating Scale scores 
(Kg., Gr. Land Gr.2) 



background information did not report the father’s education). The inclusion of father’s 
education would have greatly reduced the sample size and limited the representativeness of the 
sample. 



For the current analysis, we included only gender and age as participant characteristics. 
These two factors have often been found to influence children’s academic performance and 
behavior. In addition, in the High/Scope Perry Preschool study 1 and in the Long-Term Benefits 
of Head Start Study, 2 program benefits were found to be greater for females. Although risk 
factor information (e.g., low birth weight, low income etc.) would also represent relevant 
participant characteristics, we do not have this information for the children in the comparison 
group. Thus, these data were also not a part of the comprehensive analysis for the whole sample. 



'Schweinhart, L. J., Barnes, H. V., & Weikart, D. P. (1993) Significant benefits : The High/Scope Perry 
Preschool study through age 27. Ypsilanti, MI: High/Scope Press 

2 Oden, S., Schweinhart, L. J., & Weikart, D. P. .with Marcus, S. M., & Xie, Y. (in press). Into adulthood: 
A study of the effects of Head Start. Ypsilanti, MI: High/Scope Press 
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Nevertheless, we have included risk factors in a comprehensive analysis of MSRP children only, 
and we report the results later when we examine program effects at various study sites. 
Participants’ ethnicity is another relevant characteristic. Since this variable was not collected at 
the start of the project, we have included it in the current Parent Interview and will include it in 
the future analysis when the data collection is complete. 

Another key covariate in our analysis is the study site. Our basic descriptive analysis 
showed that there were significant differences in MSRP child outcomes across the study sites. 
Further analyses to examine how group effects differ by site found that program effects were 
significant in some sites, but not in the others. We also found that by including the site variable 
in the analysis, we could explain about 10 percent more of the total variance in child outcomes. 
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Key Outcome Variables Selected 



Child Observation Record scores at kindergarten and School Readiness Rating Scale 
scores from kindergarten through Grade 2 were included in the comprehensive analysis this year. 
Because the number of study participants with both grade retention information and covariate 
information was limited, grade retention was not included as an outcome variable in this year’s 
comprehensive analysis. However, a simple comparison of grade retention between the MSRP 
and comparison groups was conducted. Due to insufficient Parent Interview and School Records 
Review data, parent involvement, parent expectations and special education services have not 
been included for either the comprehensive analysis or other analyses to date. The continuing 
Parent Interview data collection and the data collection for the school records review in the 
previous years will help to provide sufficient data to be included in the future analyses. 

Statistical Approach 

The statistical approach we employed for this comprehensive analysis was the analysis of 
covariance, essentially analysis of variance within a regression model. This approach provides 
for an analysis of both continuous and categorical variables within the same overall model. It 
helps to examine the MSRP effects while estimating the effects of all the other covariates at the 
same time. The results of the analysis will show whether or not the MSRP has effects on the 
participants’ outcomes (e.g., COR and SRRS scores), while controlling for the potential 
influence of the other covariates in the model. 

Status of Comprehensive Analysis 

To date, the analyses of covariance have only been applied to Cohort 1 which has 
sufficient data on family socioeconomic background. Table 8 presents the Cohort 1 sample size 
for the comprehensive analysis by outcome variable. Since the comprehensive analysis requires 
the complete data for each of the covariates, any one missing covariate for a participant will 
cause the exclusion of that case from the analysis. As a result, the analysis sample size for each 
of the outcome variables was reduced, ranging between 12 to 22 percent of the total sample 
across the variables. 

As noted above, no comprehensive analysis has yet been conducted for Cohort 2, due to 
the limited number of study participants who have complete data for both covariates and 
outcomes (see Table 9). Our continuing Parent Interview data collection will help to supplement 
the missing data in family background and allow for the more comprehensive analysis for Cohort 
2 at a later stage. 
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Table 1.8. Cohort 1: Sample Size for Comprehensive Analysis by Outcome Variable 



Outcome Variables 


Sample Size 
with Some Data 


Sample Size for 
Comprehensive Analysis 




n 


% 


n 


% 


Child Observation Record (COR) 


464 


74% 


387 


62% 


School Readiness Rating Scale (SRRS) Kg. 


499 


80% 


364 


58% 


School Readiness Rating Scale (SRRS) 
Gr.l 


445 


71% 


348 


56% 


School Readiness Rating Scale (SRRS) 
Gr.2 a 


360 


58% 


265 


49% 



Note. The percentages are calculated on the basis of the total identified sample size of 623. 
a 39 participants (6%) were not included in the second grade analysis because they were 
retained in first grade. Taking the grade retention rate into consideration, the resulting 
analysis sample size is about 49% of the total sample of second grade participants. 



Table 1.9. Cohort 2 : Sample Size by Type of Data 



Grade 


Sample / Type of Data 


n 


%of 

Identified 

Sample 


Kg. 


Identified sample size 


629 


100% 




Number of participants with data (Study sample size) 


579 


92% 




Number of participants with COR 


399 


63% 




Number of participants with SRRS (Kg) 


349 


55% 




Number of participants with both 
COR and family background information 8 


288 


43% 




Number of participants with both 
SRRS and family background information 8 


271 


46% 


Gr.l 


Number of participants with SRRS (Gr.l) 


322 


51% 




Number of participants with SRRS (Gr.l) and family 
background information 


260 


41% 



Note. “Total number of Family background forms completed is 442, when we include the 
supplementary data obtained during the fall of 1999. 
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Current Findings 



We next present the major findings from the comprehensive data analysis (described in 
the previous section), which takes the covariates into account in assessing child outcomes for 
Cohort 1. The outcome variables are Child Observation Record scores (COR kindergarten) and 
the teachers’ ratings for the School Readiness Rating Scale scores (SRRS kindergarten, Grade 1 
and Grade 2). We will also report the preliminary results for Cohort l's grade retention rate, 
when most of the students were in Grade 2. 

Overall Effects for the MSRP 

The main evaluation question for the comprehensive analysis is: Does the MSRP show 
effects on child development and school readiness, after we statistically control for the other 
covariates such as participants’ characteristics, family background, and site differences? The 
following findings provide some answers to this question. They are presented for each type of 
outcome assessment. A finding is considered to be statistically significant if its probability of 
chance occurrence is less than .05 (p<. 05). 

COR Findings 

Students who had participated in the MSRP had significantly higher level of 
development on the Child Observation Record (COR) in kindergarten than students who had 
not participated in the program. After statistically controlling for the other covariates, the 
results of the comprehensive data analysis found that the MSRP group remained significantly 
higher than the comparison group in overall development on the COR. The means and 
significance levels are provided in Figure 2. This effect was also found for 5 out of the 6 COR 
subscale scores: initiative, social relations, creative representation, music and movement and 
language and literacy, but not for logic and mathematics. 
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Figure 2. Cohort 1 : COR Scores by Group (controlled for key covariates) 
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Note. ***= p< . 001 ; **= p< .01 
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Table 10 provides a simplified description of the effects of each key covariate on child overall 
development (the COR total scores) in the comprehensive analysis. 



Table 1.10. Cohort 1 Kindergarten (Kg.): Analysis of Covariance for Child Overall 
Development on the COR 



Covariate 


Variable Description 


Evidence of Significant Effects 
on Child Overall Development 


Program 


MSRP vs. Comparison 


Yes Oc.001) 


Site 


Five Sites where COR data were 
collected 


Yes Oc.001) 


Program by Site 


Compares program effects by site 


Yes 0< 001) 


Gender 


Male vs. female 


Yes (p<. 05) 


Program by 
Gender 


Compares program effects by gender 


No 


Age 


Age in months 


No 


Mother’s 

education 


Highest years of education completed 


No 


Father at home 


Father living in home vs. not living in 
home 


No 


Household size 


Number of people in the household 


No 



Note. The total sample size (N) for the analysis was 387. The amount of variance explained in the 
analysis ( R 2 ) was 20% . 



In addition to whether or not study participants had attended the MSRP (the program 
effect), site and gender were each found to have significant effects on children’s overall 
development on the COR. A significant effect was also detected for the interaction of program 
by site on child overall development on the COR, which means that the MSRP program effects 
differed from site to site (see Appendix Table 1). The program-by-site effect is discussed in 
greater detail later in this report. 

Age and family background variables selected for the analysis were found not to have any 
significant effect on children’s overall development on the COR. Here, household income was 
not included in the analysis because it did not show any effect on COR scores and would have 
reduced the explained variance by 2 to 3 percent. However, th e family background variables 
showed statistically significant or nearly significant effects on language and literacy, social 
relations, and music and movement; and age showed significant effects on the initiative and 
logic and mathematics COR subscale scores (see Appendix Table 2). 
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SRRS Findings 



Students who had participated in the MSRP were not significantly higher in school 
readiness overall on the SRRS, but were significantly higher in some areas of school readiness 
from kindergarten through Grade 2 than students who had not participated in the program. 
Table 1 1 provides the results of the analysis of covariance for child school readiness overall 
(SRRS total scores) in kindergarten, Grade 1, and Grade 2. For child school readiness overall on 
the SRRS, although the pattern of the means was higher for the MSRP versus the comparison 
group, program effects were not found to be significant in any of the three years (see Appendix 
Tables 3, 4, 5). The covariates that found consistent significant effects on child school readiness 
overall across the years were household income and gender (higher income and female study 
participants received higher scores). There were also significant site effects on school readiness 
overall in kindergarten and again in Grade 2. Although age effects were detected in kindergarten, 
the effects decreased in the following years. 



Table 1.11. Cohort 1 Kg. / Gr.l / Gr.2: Analysis of Covariance for School Readiness Overall on 



SRRS 



Covariate 


Variable Description 


Evidence of Significant Effects 8 on 
School Readiness Overall 

Kg Gr.l Gr.2 

(n=364) (n=348) (n= 265) 


Program 


MSRP vs. Comparison 


— 


— 


— 


Site 


Sites where SRRS data were collected 


Yes 


— 


Yes 


Program by Site 


Compares program effects by site 


Trend 


— 


— 


Gender 


Male vs. female 


Yes 


Yes 


Yes 


Program by Gender 


Compares program effects by gender 


— 


— 


— 


Age 


Age in months 


Yes 


Trend 


— 


Mother’s education 


Years of education that mothers 
completed 


— 


— 


Trend 


Father at home 


Father living in home vs. not in home 


— 


— 


— 


Household size 


Number of people in the household 


Yes 


— 


— 


Income 


Household income per year 


Yes 


Yes 


Yes 



Note. a Significant effects are p< 05; trends=nearly significant effects:. 10 > p >.05. For the Kindergarten (Kg.) 
analysis, the amount of variance explained ( R ?) was 26%. For the Grade l(Gr.l) analysis, R 2 = 20%. 

For the Grade 2 (Gr.2) analysis, R 2 = 20%. 
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Despite the lack of program-effect findings for child school readiness overall on the SRRS, 
the analysis did find program effects in favor of MSRP on some areas of school readiness. As 
shown in Figure 3, kindergarten teachers rated MSRP students significantly higher than the 
comparison group on retaining learning and ready to learn (the most direct way to define school 
readiness); Grade 1 teachers rated MSRP students significantly higher in good attendance ; Grade 
2 teachers rated MSRP students significantly higher in interest in school work and physical 
abilities. In addition, nearly significant trends in favor of the MSRP group were found in retaining 
learning and ready to learn in Grade 1 and social and emotional development in Grade 2 (see also 
Appendix Table 6). 



Figure 3. Cohort 1: MSRP vs Comparison for SRRS Item Scores 




MSRP group 




Comparison group 



Note. * = p<.05 




BEST COPY AVAILABLE 
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Grade Retention Findings 

Students who had participated in the MSRP had a significantly lower rate of grade 
retention than students who had not participated in the program. In addition to COR and SRRS 
from kindergarten through Grade 2, another important outcome variable is grade retention. As 
noted earlier, a preliminary analysis of this variable, without statistical adjustment for the 
covariates, was conducted. The chi-square analysis for grade information collected this year 
showed that the MSRP group had a significantly lower rate of grade retention than the comparison 
group for Cohort 1. As shown in Table 12, the percentage of grade retention for the MSRP group 
was 8 percent, compared with 15 percent for the comparison group. 



Table 1.12. Cohort 1 Gr.2: MSRP vs. Comparison Group in Grade Retention Rate 



Group 


Retained 


Not Retained 


Total 


Statistical 


n 


% 


n 


% 


Significance 


MSRP 


17 


8% 


195 


92% 


212 


p < .05 


Comparison 


24 


15% 


134 


85% 


158 


Total 


41 


11% 


329 


89% 


372 





Note. All the 41 retained students were in first grade in the 1998-1999 school year 



MSRP Program Effects across Study Sites 

Program Effects were Found at Some Sites But Not Others 

Although overall MSRP program effects were found from kindergarten through Grade 2 for 
Cohort 1, program effects were not found at every study site. This is indicated by the significant 
interaction effects of program by site. To better understand program-by-site effects, we 
descriptively compared children’s overall development on the COR at each site (see Figure 4). 

Figure 4 gives a general view of how MSRP program effects differed in the sites where the 
COR data were available. In Sites B, D and E, the MSRP groups achieved significantly higher 
overall development on the COR than their comparison groups. In Site C, children’s overall 
development for the two study groups were about the same. In Site A, the scores were in favor of 
the comparison group, although not to a statistically significant extent. Program-by-site effects 
were also found on every COR subscale score (see Appendix Table 7). 

Program-by-site effects for child school readiness overall on the SRRS are not as strong as 
for child overall development on the COR, only nearly significant trends for program-by-site 
effects were found in kindergarten (see Appendix Tables 3, 4, 5). 
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Figure 4. Cohort 1 Kg.: MSRP vs Comparison on COR Overall Development by Site 




Note. ***= /7<.00 1 . The analysis of covariance found statistically significant effects for program 
(pc.OOl), site (pc.OOl) and program by site (pc.OOl) on COR Total scores. 

Potential Causes for Program-by-Site Effects 

The results regarding program-by-site effects indicate that program effects varied from site 
to site. There may be many reasons for site differences in program effects, such as differences in 
the comparability of the two groups in some sites; the sensitivity of certain populations to the 
program effects; the sensitivity of the measures used for the program effects; differences in the 
administration of the data collection; or differences in the quality of the MSRP classrooms at 
different study sites. To date, two sets of analyses have been conducted to examine the potential 
contribution of risk factors and program quality to these program-by-site effects, as described next. 

The role of risk factors. For the first of these analyses, we compared the total number of 
risk factors per child for the 296 MSRP children from our longitudinal study sample across the 6 
study sites. As shown in Table 13, the percentage of MSRP children with 5 or more risk factors at 
Site C (where the MSRP group and comparison group averaged the same in overall development 
on the COR) was much higher than those at the other sites, 52 percent versus 4 to 34 percent. The 
analysis of statewide risk factor information for 2,324 MSRP children across the 6 sites in the same 
year confirmed the fact that the MSRP children enrolled in Site C had more risk factors than in the 
other 5 sites. (See Appendix Table 8.) 
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Table 1.13. Cohort 1: Distribution of Total Number of Risk Factors by Study Site 



Site 




Total Number of Risk Factors 






Total Number 
of children 


1-2 




3 




4 


5 or more 


n 


% 


n 


% 


n 


% 


n 


% 


Site A 


5 


16% 


7 


22% 


9 


28% 


ii 


34% 


32 


Site B 


13 


23% 


18 


32% 


11 


19% 


15 


26% 


57 


Site C 


7 


11% 


10 


15% 


14 


22% 


34 


52% 


65 


Site D 


7 


13% 


15 


28% 


16 


30% 


15 


28% 


53 


Site E 


36 


49% 


23 


31% 


12 


16% 


3 


4% 


74 


Site F 


7 


47% 


5 


33% 


1 


7% 


2 


13% 


15 


Total 


75 


25% 


78 


26% 


63 


21% 


80 


27% 


296 



Note. Chi-square analysis found a significant difference in number of risk factors 
across the 6 study sites, pc.OOl 



The analysis of covariance was then employed in order to examine the potential 
contribution of the number of risk factors on MSRP children’s overall development on the COR, 
while controlling for the potential influence of other covariates, such as site, gender, age and 
family background variables. For this analysis, only 10 risk factors that had been found to be 
significantly correlated with COR total scores (at p<.05) were selected to qualify for the number of 
risk factors. Table 14 shows that site as well as number of COR-related risk factors had significant 
effects on MSRP children’s overall development on the COR when adjusting for other covariates. 
These results show that MSRP children who had more of the identified key risk factors also had 
lower overall development on the COR. 
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Table 1.14. Cohort 1 Kindergarten (Kg.): Analysis of Covariance for MSRP Participants’ 
Overall Development with COR-Related Risk Factors 



Covariates 


Variable Description 


Evidence of Significant Effects 
on Child Overall Development 


Site 


Five sites where COR data were 
collected 


Yes (pc.OOl) 


Number of risk 
factors 


Number of COR-related risk factors 


Yes (pc.OOl) 


Gender 


Male vs. female 


No 


Age 


Age in months 


No 


Mother’s education 


Highest years of education completed 


No 


Father at home 


Father living in home vs. not living in 
home 


No 


Household size 


Number of people in the household 


No 



Note. The total sample size for the analysis was 196. The amount of variance explained ( R 2 ) was 21% . 



Table 15 lists the average number of COR-related risk factors across the 6 study sites. The 
analysis of variance found statistically significant differences in the numbers of COR-related risk 
factors across the six study sites. The average number of COR-related risk factors was the highest 
at Site C (mean=2.1 1). 



Table 1.15. Cohort 1: Average Number of COR-Related Risk Factors in Six Study Sites 



Site 


n 


Mean 


Standard 

deviation 


Statistical significance 


Site A 


32 


1.56 


1.22 




Site B 


57 


1.91 


1.07 




Site C 


65 


2.11 


1.59 


p < 001 


Site D 


53 


1.06 


1.08 




Site E 


74 


.49 


.60 




Site F 


15 


1.00 


.76 




Total 


296 


1.36 


1.28 





Note. Chi-square analysis found a significant difference in number of COR-related risk 
factors across the 6 study sites, p <. 001 



In summary, the first set of analyses showed that there were significant differences in the 
numbers of risk factors for the MSRP children at the 6 study sites. Both the total number of risk 
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factors and number of COR-related risk factors were much higher for Site C (where no apparent 
MSRP effects were found) than for the other 5 sites. Having more risk factors was found to be 
related to children having lower overall development on the COR. Thus, the results suggest that 
the higher number of risk factors for the MSRP children at Site C may explain their low overall 
development on the COR and lack of apparent program effects. However, this conclusion cannot 
be definite because the number of risk factors in the comparison group at Site C is unknown. The 
results also show the importance of controlling for risk factors in the future study design and 
analysis of program effects. 

The role of program quality. The second set of analyses was conducted to examine 
whether or not program quality may explain some site differences in program effects. For this 
purpose, a comparison of MSRP Program Quality Assessment (PQA) scores across the six study 
sites was conducted. The results are presented in Figure 5, which shows that although the overall 
quality of MSRP classes was quite high, the program quality levels were not the same across the 
sites. The chi-square analysis found significant differences in PQA total scores across the six study 
sites (see Appendix Table 9). In Sites B, D and E (where significant program effects in favor of 
MSRP were found on child development on the COR), the total scores for their PQA were 
relatively higher than for Site A and Site C, where no program effects were detected. 

If we take a further look at the results shown in Figure 6, we find that in Site A (where no 
program effect was found), the subscale scores for curriculum, instructional staff, and philosophy 
were the lowest, compared with the other subscale scores of program quality. These results 
suggest that program quality may provide an additional explanation for program-by-site effects. 

The results also suggest the importance of further development of the MSRP at some sites. 



Figure 5. Cohort 1 : Overall Program Quality on the PQA by Site 
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Figure 6. Cohort 1:Site A vs. Ail other sites for PQA Total & Subscale 
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Summary of the Major Findings 

This year we conducted a comprehensive analysis to test for Cohort 1 MSRP program 
effects over and above the effects of key background variables. The results of this analysis showed 
strong program effects for child development on the COR at kindergarten and significant program 
effects for some areas of school readiness on the SRRS from kindergarten through the second 
grade. Further, preliminary analysis found that the Cohort 1 MSRP group had a significantly lower 
grade retention rate than the comparison group (8% versus 15%) by the end of Grade 2. These 
results indicate an overall program effect on child development and readiness for school 
participation. However, at the same time, the analysis revealed a significant variation in program 
effects across the study sites. Thus, program effects were found in some sites, but not in the others. 
While there are many possible methodological reasons and potential risk factor differences to 
explain the program-by-site effects, the significant differences in the program quality across the six 
study sites suggest a need for further development of the MSRP at some sites, especially in 
curriculum, instructional staff, and philosophy. In addition, an analysis of risk-factor effects on the 
MSRP children’s developmental outcomes indicated that MSRP children who had more risk 
factors had lower development on the COR, which suggests the importance of taking the risk 
factors into account if possible, in the future evaluation design. 
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Data Collection Plan for the Next Two Years 



This section presents the data collection plan for the next two years. There are two major 
innovations: (1) include Michigan Educational Assessment Program (MEAP) scores in Cohort l's 
fourth-grade data collection; (2) begin a new cohort in the 2000-2001 school year, in order to 
continue to capture evidence of recent MSRP effects. 

Plan for 1999-2000 



In 1999-2000, we are focusing on data collection for Cohort 1 Grade 3 outcomes and 
completion of the previous years’ data in Parent Interview and school records review. The data 
collection for Cohort 2 will end due to large amount of missing data. The data to be collected are as 
follows: 

Cohort 1 

• School Readiness Rating Scale data (Grade 3) 

• School records review data (Grade 3), including any missing cases for kindergarten, Grade 
1 and Grade 2 

• Completion of the Parent Interview data collection for all cases 

• Securing permission for MEAP score data collection in Grade 4 

Cohort 2 

• Completion of the Parent Interview data collection for 40 additional families 

• School records review data for all cases (Grade 2) 

Plan for 2000-2001 

In 2000-2001, a new cohort will be initiated, and Cohort l's Grade 4 outcomes, including 
MEAP scores, will be collected. Following are the data to be collected: 

Cohort 1 

• School Readiness Rating Scale data (Grade 4) 

• School records review data (Grade 4) 

• MEAP scores (Grade 4) 

Cohort 2 

No further data collection 




1-29 



42 



Cohort 3 (new cohort) 

• PQA data in the MSRP year 

• COR data in the MSRP year 

• Family Background Questionnaire 

• School records review data in the MSRP year 

• Parent Interview data in the MSRP year 
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Part 2 - Statewide Data for Program Quality and Risk Factors 

Each year, all MSRP grantees must complete the State-mandated Program Quality 
Assessment for all classrooms. In addition, each MSRP grantee must provide risk factor data for 
each child in the program. As part of its MSRP program evaluation responsibility, High/Scope 
aggregates and analyses two additional statewide data sets concerning Program Quality and Risk 
Factors. Each data stream and summary findings are described below. 

Overview of the Program Quality Assessment 

In recent years, research in early childhood education has repeatedly linked improved 
child-level outcomes to the quality of preschool programs. The Program Quality Assessment 
(PQA) was designed as practitioner-friendly classroom quality assessment tool for use in the 
MSRP. It was developed for grantee staff to assess compliance with the Michigan State Board 
of Education’s Standards of Quality and Curriculum Guidelines for Preschool Programs for 
Four- Year-Olds (1987). The PQA is comprised of 72 items in 7 areas: Learning Environment, 
Daily Routine, Adult-Child Interaction, Curriculum Planning and Assessment, Parent 
Involvement and Family Services, Staff Qualifications and Staff Development, Program 
Management. 

Analysis of the Statewide 1998-99 Program Quality Data 

The PQA is completed for every MSRP classroom in the state. In 1998-99, PQAs were 
completed for 940 classrooms across the state. Tables 2.1 and 2.2 provide mean scores and 



Table 2.1. 1998-99 Program Quality Scores by Quartiles: Classroom Level 



Scores 


Learning 

Environment 


Daily Routine 


Adult-Child 

Interaction 


Curriculum 
Planning and 
Assessment 


Parent 

Involvement and 
Family Services 


Mean 


4.49 


4.45 


4.71 


4.30 


4.65 


75 th Percentile 


4.78 


4.83 


5.00 


4.80 


4.90 


50th Percentile 


4.55 


4.58 


4.83 


4.40 


4.80 


25 Percentile 


4.22 


4.17 


4.00 


4.00 


4.50 



scores by quartile across the seven PQA categories. Results are presented in two tables to reflect 
the division between PQA items that refer to classroom staff level responsibilities and those that 
rest with program administrators. The basic message in these results is that classrooms across 
the state demonstrate extremely high quality ratings. 
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Table 2.2. Program Quality Assessment Scores by Quartile: Program Level 



Scores 


Staff qualifications and 
staff Development 


Program Management 


Mean 


4.38 


4.56 


75 th Percentile 


4.71 


4.82 


50 th Percentile 


4.43 


4.64 


25 th Percentile 


4.14 


4.36 



Low Scoring Items 

Despite the high average scores described in the two tables above, PQA results can be 
disaggregated in such a way as to suggest areas for program improvement such as breaking out 
individual items on which a large percentage of classrooms scored poorly. 1 In this case a poor 
score is one of 3 or less. Table 2.3 (See page) presents those PQA items on which 10% or more 
of statewide classrooms scored a 3 or less. 

Several program-level PQA items show particular weakness - professional organization 
affiliation, ongoing professional development and continuity of trainer. The weakness of these 
items suggests as professional organizations, MSRP programs could be strengthened. The 
organizational weakness may be further reflected in the items related to staffing and staff 
development. Such problems are not uncommon within the early childhood community in 
general but seem especially pronounced within MSRP. 

The lack of outdoor space and outdoor play equipment stands out as a facility-related 
problem in many MSRP sites and may be a result of housing such programs in locations that 
were not designed for young children’s outdoor activities such as churches and utility spaces. 
Even public school locations can suffer from the lack of suitable outdoor play space and 
equipment specifically for preschool children. 

The cluster of weak PQA items relating to child planning, recall and small-group time 
strongly reflect curriculum issues that go to the core of appropriate practices for early learning 
programs. Despite their benefits for both learning and classroom management, child planning for 
free-choice activities and the use of recall of free-choice experience as a reflective and language- 



’The PQA was designed as a set of program quality standards - and for this reason operates from different 
assumptions than most measurement tools. Rather than looking for variation over a set of quality measurement 
indices, the “goal” in using the instrument is to score “5" on all of the items - a situation of uniformity that requires 
tailoring of statistical analyses. 
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development strategy are too often overlooked in classroom practice either because of the time 
they consume or because they are associated with particular curriculum models - or both. 

The relatively low score for anecdotal note-taking is reflective of the fact, noted in the 
analysis of the Narrative Summary reports (see Part 3), that the large majority of MSRP programs 
are using child assessment tools that do not involve extended observation and note taking. 

The incidence of these low-scoring PQA items, indicates an as yet unmet need for 
curriculum and program training and technical assistance among MSRP programs across the 
state. 

Risk Factor Data from the MSRP Statewide Evaluation 

MSRP serves four-year-old children “at risk of becoming educationally disadvantaged 
and who may have extraordinary need of special assistance.” Children are eligible to participate 
in MSRP if they are determined to have at least two risk factors (see Table 2.4) identified as a 
potential threat to later educational success by the Michigan Department of Education. Fifty 
percent of all eligible children must have low income as one of their risk factors. Low income is 
defined as federal reduced-lunch eligibility which is 185% of the federally defined poverty line. 
(Compare with Head Start eligibility at the poverty line.). Although two factors qualify children 
for access to the program, grantees are required to report the presence or absence all 25 identified 
risk factors for each child. 

When the risk factors were first introduced as program eligibility criteria, factor 
definitions were left fairly open to allow for local flexibility. Now, after several years of 
implementation, the Department of Education and High/Scope plan to survey programs identified 
as being proactive in their use of the risk factors to compile empirical definitions of how the risk 
factors are defined and substantiated in the field. In addition to this compilation of widely and 
successfully implemented factor definitions, High/Scope will make training available to the 
statewide workforce, under the GESP project, on using risk factors to help guide specific 
interventions with the at risk children in the classroom. 




2-3 



46 



Table 2.3. PQA Items on Which Over 10% of Classrooms Score 3 or Less 



PQA Categories 


PQA Item 


% of classrooms 
scorina 3 or less 


Learning Environment 


Outdoor space, equipment, materials 


22.9 




Organization and labeling of materials 


11.4 




Diversity related materials 


16.6 




Displays of child-initiated work 


13.8 


Daily Routine 


Time for child planning 


22.5 




Time for child recall 


38.0 




Small group time 


25.6 




Large group time 


12.4 




Choices during transition times 


12.2 




Outside time 


. 18.7 




Support for non-english speakers 


11.3 


Curriculum Planning 


Curriculum model 


15.8 


and Assessment 


Team teaching 


19.7 




Anecdotal note taking by staff 


30.6 




Use of child observation measure 


12.3 


Parent Involvement 


Parents on the policy making committee 


25.8 


and Family Services 


Extended learning at home 


10.3 


Staff Qualifications 


Program director's education 


13.1 


and staff development 


Instructional staff's education 


11.6 




Head teacher's experience 


14.0 




Assessment to identify training needs 


17.8 




Ongoing professional development 


21.5 




Inservice specific to child development 


13.7 




Inservice combining theory and practice 


12.6 




Continuity of trainer 


29.1 




Particpatory hands-on inservice 


11.9 




Opportunities for reflecting and sharing 


14.6 




Observation and feedback 


17.1 




Supervision of supplementary staff 


17.2 




Professional organization affiliation 


42.5 


Program Management 


Continuity of instructional staff 


15.2 




Accessibility to those with disabilities 


13.2 




Funds to equip and maintain classroom 


10.4 




Funds to hire appropriate staff 


24.4 




Funds to support staff development 


16.8 




Funds to support parent involvement 


13.8 



Analysis of the Statewide 1998-99 Risk Factor Data 

Findings from the following analysis of the 1998-99 statewide risk factor data are 
presented at the state level of aggregation. Although the distribution of factors does vary by site, 
preliminary analyses have not demonstrated large differences in the risk factor profile at the 
county level or when programs are grouped by size. Tables 2.4 and 2.5 both present data over 
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time for the percentage of the MSRP population with a factor (prevalence) and the number of 
factors that 

children experience (incidence). Both prevalence and incidence demonstrate remarkable stability 
over the period for which data are presented. 



Prevalence of Risk Factors 

In 1998, risk-factor data were collected for over 19,000 children across the state. Column 
2 of Table 2.5 presents the risk factor prevalence for 1998. “Low family income” and “Single 

Table 2.4. Percentage of Children Experiencing Certain Risk Factors 2 



(1996 and 1998 MSRP data) 



Risk Factors* 


Percentage of 
Children in 1998 
(total N=18831) 


Percentage of 
Children in 1996 
(total N=19435) 


Change in 
Rank 


Low Family Income 


67.2% 


67.6% 


0 


Single Parent 


41 .6% 


41 .0% 


0 


Family History of Academic Failure 


31 .4% 


25.1% 


1 


Rural or Segregated Housing 


25.2% 


28.6% 


-1 


Unemployed Parent/Parents 


24.6% 


24.2% 


0 


Teenage Parent 


21 .4% 


20.2% 


1 


Low Parent/Sibling Educational Attainment or Literacy 


19.1% 


20.6% 


-1 


Parent/Sibling Loss by Death or Divorce 


18.7% 


17.6% 


0 


Language Deficiency or Immaturity 


18.4% 


17.3% 


0 


Developmental^ Immature 


16.5% 


15.2% 


0 


Diagnosed Family Problems 


15.4% 


12.6% 


1 


Family Density 


14.6% 


13.2% 


-1 


Nutritionally Deficient 


13.4% 


12.3% 


1 


No Stable Support System or Residence 


13.1% 


12.6% 


-1 


Substance Abuse or Addiction 


12.6% 


1 1 .4% 


0 


Chronically III Parent or Sibling 


12.0% 


10.6% 


1 


Child's Long Term or Chronic Illness 


1 1 .5% 


10.8% 


-1 


Low Birth Weight 


11.1% 


9.4% 


0 


Limited English Speaking Household 


10.5% 


8.5% 


0 


Destructive or Violent Temperament 


8.3% 


7.7% 


0 


Family History of Delinquency 


7.5% 


6.0% 


1 


Incarcerated Parent 


7.2% 


6.0% 


1 


Child Abuse or Neglect 


6.8% 


6.8% 


-2 


Diagnosed Handicapping Condition 


5.4% 


5.2% 


0 


Other 


2.9% 


3.4% 


0 



’ Excludes those with less than 2 risk factors. 



9 

Since two risk factors is a minimum required for MSRP participation, cases with less than two risk factors 
indicated were considered erroneous and were excluded from the analysis. They amounted to less than 2 % of all 
cases. 
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parent” are the most common risk factors: about 75% of all children in the MSRP program have 
one of these two factors. This is as expected in a program that requires that 50% of all eligible 
children have low income as one of the two minimum required factors. However, only about 
32% of MSRP children have both factors and none of the risk factors correlate more strongly 
than “Low family income” and “Single parent.” Only 7 pairs of factors have Pearson correlation 
coefficients greater than .20 and none are greater than .25. 

Risk Factor Incidence 

MSRP children averaged 4.19 risk factors per child. More complete risk factor incidence 
information is presented in Table 2.5 below. Incidence is a crucial consideration because logic 
suggests that the more factors a child demonstrates, the higher the risk of school failure. As 
reported in section I of this report, data from the MSRP State Longitudinal Study reinforces this 
intuition. For the cohort-one MSRP group of 

Table 2.5. Risk Factor Levels 



Number of Risk 
Factors* 


Percentage of Children in 
1998 (total N=18831) 


Percentage of Children in 
1996 (total N=19435) 


2 


22.4% 


22.4% 


3 


25.8% 


26.5% 


4 thru 7 


44.1% 


42.0% 


8 or more 


7.7% 


9.1% 


Total 

* r~ i. . j xi :xi_ 


100.0% 


100.0% 



* Excludes those with less than 2 risk factors. 



the State Longitudinal Study, the number of risk factors documented in preschool is inversely 
correlated child outcome measures in the kindergarten year. This relationship holds up in multi- 
variate regression analyses as well. 

Levels of Risk: High and Low 

Because findings from the MSRP State Longitudinal Study have suggested that higher 
numbers of risk factors are associated with lower child development outcomes, we thought it 
would be useful to conceptualize the statewide risk factor profile in terms of high and low risk. 
When the line is drawn between the roughly half of the MSRP population with 2-3 risk factors 
(>2=9,307) and the remaining half of the population with 4 or more factors (9,712), two very 
distinct populations appear. These population segments are described in Table 2.6 below. 
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Table 2.6. Top 10 Risk Factors by Level of Risk (1998 MSRP data)* 



Risk Factors for Low Risk (2-3 factors) 
Children (N=9307)** 


%of 

Cases 


Risk Factors for High Risk (> 3 factors) 
Children (N=9712)** 


%of 

Cases 


Low Family Income 


58.0% 


Low Family Income 


74.3% 


Single Parent 


31 .7% 


Single Parent 


48.7% 


Rural or Segregated Housing 


17.8% 


Family History of Academic Failure 


44.3% 


Family History of Academic Failure 


15.7% 


Unemployed Parent/Parents 


33.5% 


Unemployed Parent/Parents 


13.3% 


Teenage Parent 


30.3% 


Language Deficiency or Immaturity 


12.1% 


Rural or Segregated Housing 


29.7% 


Developmentally Immature 


10.7% 


Low Parent/Sibling Educational Attain- 




Teenage Parent 


10.1% 


ment or Literacy 


28.4% 


Limited English Speaking Household 


9.7% 


Parent/Sibling Loss by Death or Divorce 


27.1% 


Parent/Sibling Loss by Death or Divorce 


8.2% 


Diagnosed Family Problems 

Language Deficiency or Immaturity 


24.0% 

23.0% 



* Factors in bold are those not present in the corresponding population with the different 
level of risk. 

** Excludes those with less than 2 risk factors. 




2-7 



50 



Part 3 - Grantee Evaluation Support Project 



Summary of the Grantee Evaluation Support Project 

High/Scope Foundation has a long history of work and research in early childhood 
education. In particular, the High/Scope Perry Preschool study 1 , begun in 1962 continues to 
demonstrate that high quality early childhood programs can have long-term positive impacts on 
the lives of disadvantaged children by improving their educational attainments, reducing their 
involvement in serious crime, increasing their lifetime earnings, and reducing their dependence 
on social services. The result is that high-quality early childhood programs can be cost-beneficial, 
returning to the state many dollars in increased tax revenues and reduced social service and 
criminal justice costs for each dollar invested in running the program. 

High/Scope was, therefore, an early supporter of the State of Michigan’s plan, in the last 
15 years of the 20 lh century, to launch a state-wide preschool program that would bring high- 
quality early childhood services to unserved, at-risk children throughout the state. High/Scope 
staff provided evidence, in legislative hearings, of the value and economic sense of good early 
childhood services as the MSRP program was formulated, and High/Scope sought to play an 
active role in the program’s implementation. 

Knowing the importance of quality to the success of early childhood programs and being 
familiar with the contribution that ongoing evaluation makes to assuring high quality, 

High/Scope began in the early 1990s to discuss with the Michigan Department of Education and 
with the W. K. Kellogg Foundation how we might assess and contribute to the quality of the 
MSRP programs throughout the state. High/Scope pursued funding from both the State 
Legislature and from the Kellogg Foundation to support both an evaluation and a training effort 
in the state. In April 1995, in response to a request for proposals from MDE, High/Scope 
received a grant to begin planning a state-wide evaluation of the MSRP project. Subsequent 
MDE funding allowed the evaluation plans to be put into action in late 1995. High/Scope also 
began at that time to formulate a state-wide training project that would enhance the capacity of 
local MSRP projects to gather and use evaluation information. In 1998 High/Scope received a 
grant from the W. K Kellogg Foundation to support the training effort designed to help local 
MSRP projects use evaluation to support the quality of their programs. The resulting set of 
activities, described in detail below, constitutes the High/Scope MSRP Grantee Evaluation 
Support Project (GESP). 

Goals and Plan of the MSRP Grantee Evaluation Support Project 

The goal of the GESP is to saturate School Readiness programs throughout the state with 
an evaluation consciousness that encompasses both organizational and technical underpinnings 



'Schweinhart, L. J., Barnes, H. V., & Weikart, D. P. (1993) Significant benefits: The High/Scope Perry 
Preschool study through age 27. Ypsilanti, MI: High/Scope Press 
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of successful local evaluations. In addition to expertise in data collection and feasible evaluation 
designs, the GESP seeks to enhance the capacity of local MSRP programs to use evaluation data 
in a process of ongoing program improvement. It provides specific training in how local 
programs can use evaluation results (both local and state-wide) to strengthen and improve the 
particular features of their local programs so that they can achieve high levels of program quality 
effectiveness. 

Figure 3.1, (See next page.) depicts the planning, decision making structure at the local 
level - the arena in which performance indicators can play a role in affecting the actions local 
programs undertake in pursuit of overall program goals. This is the operational focus of the 
feedback loop that effective evaluation can provide. Figure 1 also puts local evaluation in a 
somewhat realistic context of where these processes are seen as parallel to local project’s main 
theme of operating and providing services. 

The Grantee Support Project includes components that specifically address the challenges 
and skill needs for enhancing the evaluation consciousness of local early childhood programs. 
These components are taken up in the description of the Grantee Support Project training 
activities, that follows. 

Training and Support Activities of the Grantee Evaluation Support Project 
Linking Assessment to School Improvement 

The linkage of assessment to change at the classroom/instructional level is not self- 
implementing, but rather requires that programs build in an ongoing process of using assessment 
information to guide program change and staff development. Studies of public school 
assessment 2 indicate that: 

• Classroom-level change requires a substantial and long-term commitment to 
professional development 

• The assessment-improvement linkage is not successfully made without 
administrative support 

• Assessment for accountability or reporting to high levels of authority may not be 
appropriate for purposes of instructional improvement 

• There is a need to work out specific linkages between assessment results and 
corresponding strategies for program improvement 

The MSRP Grantee Evaluation Support Project builds upon the school reform concept to 



2 Schmoker, Mike. 1996. Results: The Key to Continuous School Improvement; Valencia, Sheila and Karen 
Eixson. 1999. “Policy Oriented Research on Literacy Standards and Assessment” CIERA Report #3-004; Stecher, 
Brian, Sheila Barron, Hilda Borko, and Shelby Wolf. 1997. “Important Features of State Assessment Systems from 
the Local Perspective.” CSE Technical Report 472; Noble, Audrey and Mary Lee Smith. 1994. “Measurement 
Driven Reform: Research on Policy, Practice and Repercussion.” CSE Technical Report 381. 
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move from measurement instruments and comparison data to the delivery of support and training 
tailored to individual program improvement needs. In addition, school reform research above 
identifies the issues of low- vs. high-stakes evaluation and organizational structure as 
considerations for training to address if it is to have an impact on programs and instruction and 
hence, on subsequent assessment results. 

Focusing on Low-Stakes, Program Improvement Evaluation 

We see the primary purpose of the GESP as supporting local programs in using their local 
evaluations for formative, low-stakes decision making about program improvement. Our goal is 
to assist local programs in using their evaluations for program improvement. The strategy of the 
GESP is to focus workshop training on the “Goals/ Activities/Outcomes” section of the Narrative 
Report Summary so that this section embodies the MSRP local evaluation and also serves as a 
working document that informs on-going program improvement. The Narrative Report 
Summary, is the section of the local program’s annual report to MDE in which they are asked to 
describe their program’s goals, the activities used to implement these goals, and the evaluation 
results reflecting the status of these efforts. Using the Goals/Activities/Outcomes section of the 
Narrative Summary this way allows local evaluations to serve both a monitoring/reporting 
function with respect to state authorities and a program improvement function locally. 

Two Tiers of Training 

The GESP training strategy targets the two tiers of persons involved with program 
evaluation at the local level and stems from the recognition that useful evaluation involves 
organizations, not just individuals. Although it is not, in most medium-sized and larger districts, 
the classroom staff that carry out program evaluation and reporting responsibilities, it is 
classroom staff who collect the data on program quality, parent involvement, and child 
development. Furthermore, classroom staff need to be comfortable interpreting evaluation 
findings after assessments of individual children and program attributes are aggregated into the 
data on program-level findings. 

The second target in the evaluation capacity-building effort are those who actually make 
evaluation decisions, such as choosing assessment instruments, assembling and analyzing data, 
and reporting to the state at the grantee level. An analysis of these reports (see Part 2) indicates 
that about 60% of those signing the Narrative Summary Reports in 1997-1998 were persons 
other than MSRP teachers or education specialists. It is the Narrative Report signers that we hope 
to reach through the second tier of training - Individualized Evaluation Consultation Sessions. 

Training Activities of the Grantee Evaluation Support Project 

As was stated earlier, the overall mission of the GESP is to strengthen the capacity of 
School Readiness program staff throughout the state to evaluate the quality of their programs and 
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how well their programs contribute to children’s development. In order to accomplish these ends, 
several GESP objectives were established: 

• Increased local understanding of evaluation of program quality and 
effectiveness 

• Increased use of comprehensive observational child assessment instruments 

• Increased local capacity for high-quality evaluation 

• Increased linkage between program evaluation and program improvement 

• Identification of a cohort of evaluation trainers/consultants based in grantees and 
intermediate school districts 

In order to meet these objectives, three workshops were developed that would address 
important components of the Local Evaluation Model: (a) assessment of the process of what the 
program does and the services it provides, (b) assessment of child outcomes, and (c) a 
methodology for local evaluations emphasizing measurement, instrumentation, and the use of 
comparison data. For the training cycle September 1998 to August 1999, the workshops topics 
were: 



Using the High/Scope Program Quality Assessment (PQA) to Assess Program 
Quality: a two-day workshop focusing on program quality guidelines and 
assessment. It features the High/Scope Program Quality Assessment instrument, 
used by the Michigan Department of Education to monitor program quality and 
compliance of all MSRP programs. 

Using the High/Scope Child Observation Record to Assess Children’s 
Development - a two-day workshop presenting major features of observational 
child assessment using the High/Scope Child Observation Record. 

Conducting a Preschool Evaluation - a one-day evaluation design workshop 
linking measurement instruments, data collection, comparison data from statewide 
evaluation results, and the reporting of local evaluation results. 

The Program Quality Assessment and Child Observation Record Workshops provide 
valuable knowledge and skill for the Michigan early childhood workforce and are central to the 
MSRP Local Evaluation Model. The Evaluation Design workshop helps local evaluators think 
about their programs in terms of two distinct types of evaluative activity: process evaluation and 
outcome evaluation. Process evaluation involves descriptions of program attributes and 
measurements of their strength using the Program Quality Assessment (PQA) and Parent 
Involvement Questionnaire (PIQ). Outcome evaluation focuses on gains in child development 
derived from scores on the Child Observation Record (COR) and the School Readiness Rating 
Scale (SRRS). 
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Workshop evaluations and trainer reports indicate that the 1998-99 training cycle has 
produced some significant results. One result is that the GESP workshops have had an impact on 
participants’ general knowledge of best practices in the field of early childhood education. COR 
trainings have had an impact on participants’ understanding of key areas of child development 
and their ability to collect child assessment information. The PQA workshops have emphasized 
best practices in the areas of adult-child interaction, preschool learning environment, and daily 
routine and improved participant understanding of them. 

Many participants of the Conducting a Preschool Evaluation workshop reported in their 
evaluations that they came away from the training with a new understanding of the importance of 
using comprehensive child assessment instruments that are observation-based in order to provide 
a thorough and authentic record of children’s development over time. Another outcome is that, 
by becoming familiar with the assessment instrumentation used by the State Longitudinal Study 
(i.e., COR, PQA, PIQ, and SRRS) participants gained a clearer understanding of how school 
readiness is defined and measured in the State Longitudinal Study. Additionally, they gained a 
broader perspective on school readiness as it spans preschool through the early elementary 
grades. Finally, many participants expressed an interest in a using a computerized version of the 
COR in order to streamline their workload and professionalize the reporting forms they provide 
to parents. 

In response to what was learned in the 1998-99 training cycle, four additional workshops 
were offered in the 1999-2000 training cycle: 

Improving Program Quality: A PQA Follow-up - a one day workshop that 
focuses on best practices in the area of preschool active learning and adult-child 
interaction strategies and their impact on these sections of the PQA: Learning 
Environment, Daily Routine, and Adult-Child Interaction. This workshop starts to 
bridge the gap between evaluation training and curriculum training. 

Using the Computerized COR - a one-day workshop that provides a thorough 
introduction to using the computerized version of High/Scope’s COR. The 
computerized version is less time-consuming than the paper version. 

Observation and Assessment of Young Children - a one day workshop that 
emphasizes the importance of using observation and authentic assessment to 
understand children’s development; offers participants guidance in selecting and 
developing a plan for using a child development instrument that is both 
comprehensive and diagnostic. This workshop was designed for Detroit and other 
grantees who are using non-observational instruments at present. 

Individual Evaluation Consultation Session: A follow-up to the Evaluation 
Design Workshop - a half day session that offers individualized assistance and 
custom local evaluation data reports to individuals who are at various stages of the 
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process of designing or conducting an evaluation of their local program. The 
individual consultations go beyond the evaluation design workshop to work through the 
local issues with selected grantees. 

Summary of GESP Training Participation 

For the 1998-99 MSRP training cycle (September 1998 to August 1999), the GESP 
offered a three-workshop series statewide. In the 1999-2000 cycle (August 1999 to January 
2000), the GESP has offered four additional workshops statewide, and has delivered training to 
1688 participants at over 70 workshops and conferences over its two years. Table 3.1 out all the 
workshops so far conducted by type, and number of participants. The workshops were hosted 
primarily by Intermediate School Districts across the state, with 50 of 57 ISD’s hosting. 



Table 3.1. GESP Trainings Delivered by Workshop Type 





Conducting a 
Preschool 
Program 
Evaluation 


PQA 


COR 


Computer 

COR 


PQA 

Follow 

Up 


Observational 

Assessment 


individual 

Consultation 


# of 


23 


28 


33 


5 


4 


l 


4 


workshops 
















# of 


269 


486 


628 


101 


121 


60 


23 


trainees 
















% of total 


(16%) 


(29%) 


(37%) 


(6%) 


(8%) 


(3%) 


0%) 



Table 3.2. Trainings Delivered vs. Potential Demand 





Trainings delivered 
1998-2000 


Potential demand 


For ISDs in which 


1688 


3339 


workshops occurred 






For all ISDs 


1688 


4323 



Training Coverage 

The distribution of trainings across the various categories of the statewide MSRP 
workforce, or coverage, can be considered as the ratio of trainings delivered to potential demand 
at district, ISD, and state levels. The potential demand for training among the MSRP workforce 
was estimated from the total number of MSRP Children and the approximate adult/child ratios 
across the state. The estimate of the total MSRP staff statewide times the number of workshops 
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gave us an approximation of the potential demand for training. In the 1998-2000 trainings so far, 
High/Scope has trained a number equivalent to half of all the MSRP staff in the ISD’s that held 
workshops (although not all participants were from MSRP, see Table 3.2) 



Table 3.3. GESP Workshop Participants by Agency 



Agency Type 


% of total (n= 228) 


Public School MSRP 


65% 


Head Start 


24% 


Private Non-profit 


9% 


Private For-profit 


1% 



Table 3.4. GESP Workshop Participants by Job Title 



Job Title 


% of total (n= 2 1 1 ) 


Lead Teacher/Teacher 


56% 


Assistant Teacher 


12% 


Aide 


20% 


Director 


11% 



Characteristics of GESP Training Participants 

The following profile in Tables 3.3, 3.4 and 3.5 is based on available data from the GESP 
workshops. Tables 3.3 - 3.5 show the diversity and complexity of program type, education levels 
and job responsibilities, in MSRP workshop participation. The GESP is well-positioned to affect 
the larger field of early childhood education statewide with its breadth and depth. 

Program Characteristics Related to Assessment and Evaluation 

The assumption of responsibilities for assessment and program quality evaluation within 
MSRP grantees is extremely varied both across all agencies and when agencies are grouped by 
size. There is no effective means to predict who will be completing program quality assessments, 
child assessments, or program evaluations. This fact is extremely important given that all of the 
GESP workshops aim towards an evaluation model which depends on having the person actually 
carrying out the program evaluation and reporting to the state also making decisions concerning 
measurement, instrumentation, and evaluation design. These people must be identified agency by 
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Table 3.5. GESP Workshop Participants by Education Level 



Education 

Level 

n=251 


Less than 
HS 


HS 


<2 yrs. 
college 


Associates 

degree 


Bachelors 

Degree 


Masters 

Degree 


Education 

Specialist 


% 


3% 


8% 


14% 


9% 


39% 


16% 


8% 



agency - and are often not MSRP teachers, nor the education specialist affiliated with the 
program, nor the district contact person. In 1999-2000, only 39% of those completing the MDE 
Narrative Summary form were listed as MSRP teachers, Education Specialists, or both (see 
Figure 3.5 below). 

Training Impact 

For the 1998-1999 year, MSRP programs were asked to complete a section on the 
Narrative Summary form that elicited feedback from people who attended the training as to its 
impact on their program. Of the 350 programs that gave feedback, 37% offered some response to 
this inquiry (quite similar to the 39% estimated earlier). Table 3.6 show the range of responses 
for those who mentioned some impact of the training. 

Table 3.6. Impact of the MSRP Evaluation Workshop Series on the MSRP Workforce 



Training benefit 


% Responses 


Increased knowledge of child assessment 


38% 


Increased knowledge of program quality 


24% 


Opportunity for Staff Reflection 


8% 


Increased understanding of Evaluation 


30% 


Capacity/Program Improvement 





Monitoring the Need for Evaluation Support to MSRP Grantees 

In addition to the training-and-technical-support function which the GESP is designed to 
serve, the project also manages three program-related data streams for the state: (1) year-end 
program summaries for each grantee (Narrative Summary form), (2) Program Quality 
Assessment results for each MSRP classroom, and (3) risk factor information for each child 
enrolled in the program. The responsibility for aggregation and reporting on these large, 
statewide data sets adds program monitoring and evaluation feedback dimensions to the GESP. 

The Narrative Summary report, completed annually by each MSRP program, provides an 
excellent source of insight into the evaluation consciousness and capacity of MSRP grantees. 
The Narrative Summary forms are completed by August of each year and summarize program- 
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level information concerning demographics, participant transitions, parent involvement, program 
strengths and weaknesses, enrollment, and evaluation of program processes and outcomes. 
Several questions in the report form ask the program to describe evaluation results and are useful 
in understanding what programs choose to evaluate and how well they are able to do it. 

Completion of the Narrative Summary represents a significant amount of effort. The 
GESP goal is to make the process of completing the Narrative Summary form an integral part of 
a program planning and improvement process within each MSRP local program. Because the 
form requires input from project staff as well as administrators, it is a perfect opportunity to 
engage the entire organization in a process of review, goal setting, and evaluation planning. 

Analysis of the Narrative Summary Form as an Indicator of Evaluation Capacity 

This section presents an estimate of the evaluation capacity of MSRP grantees from our 
analysis of the Narrative Summaries submitted by local MSRP programs for the 1998-99 
program year. The estimates are in four areas - assessment instruments used, use of child-level 
outcomes in program evaluations, inclusion of longitudinal effects, and organizational capacity to 
carry out program evaluation. 

State program evaluation guidelines require the grantees have in place an evaluation plan 

that: 



covers implementation of all required program components and an assessment of 
gains of the participating children in the program. These gains shall reflect social 
relationships, emotional development, physical coordination as well as cognitive 
growth and include a process for collecting longitudinal data on children 
participating in the program as they enter kindergarten and attend first grade. 

(State Board Criterion) 

Overall, however, our analyses indicate that MSRP grantees statewide are not focused on 
the assessment of child-level outcomes, are not experienced in the translation of child assessment 
into larger judgements about program processes and strategy, and are located within education 
organizations which often lack a structure of responsibility for assessment and evaluation which 
integrates preschool staff, early elementary teachers and school administrators. 
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Comprehensive Child Assessment 

One of the key issues in the measurement of child-level outcomes is the type of 
instrument used. This is especially relevant given the state requirement that programs measure 
fall-to-spring gains in a broad array of content areas (e.g. social development, emotional 
development, etc.). As Figure 3.2 below indicates, 30% of MSRP grantees report using child 
outcome measurement instruments which are comprehensive in scope and have psychometric 
properties making them able to measure gain. Another, 17% of all program use one of the 
commercial screeners (e.g. Dial-R, Brigance, Gessell, ABC, and Early Screening Inventory.) 
These are often instruments with established reliability and validity but are not typically 
appropriate for the calculation of fall-to-spring gains and are usually not comprehensive in the 
content areas measured. Finally, over 50% of public school preschool programs use a child- 
outcome measure which they describe as locally developed. Many of these instruments are very 
rudimentary checklists and one-time skills inventories which cannot by themselves measure 
children’s development. 



The essential point here is that for most public-school based preschools in Michigan, 
child-level outcomes are often assessed with tools of questionable psychometric quality that do 
not meet state guidelines for measuring developmental gains. Further, the large “locally 
developed “ category in Figure 3.2 effectively precludes the comparison of outcomes across the 
majority of programs in the state. Hence, our first conclusion - that most MSRP evaluations are 

Figure 3.2 



Child Outcome Measures: MSRP 1998-99 




s 

□ 



Comprehensive 
Commercial screener 
Locally developed 



not focused on child-level outcomes because the child assessment instruments they are using are 
not suited to such a focus. 
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From Child Assessment to Outcome Evaluation 



The information provided in Figures 3.3 and 3.4 below describes the characteristics of 
evaluation results which MSRP programs’ Narrative Summaries report for child-level outcomes 
during and after the preschool year. In Figure 3.3, the second (and tallest) bar reveals that, while 
many programs effectively describe program processes and implementation, the majority of 
programs in the state do not even mention the concept of a child level outcome in their reports to 
the state. The third bar describes programs that at least mentioned that child outcomes were in 
fact occurring as a result of the program intervention. Only 10% of programs (the fourth bar) 
chose to report child outcome data that included (1) any kind of quantitative results from a (2) 
child measurement instrument and (3) relate these results to some form of meaningful evaluative 
comparison. Any two of these criteria were accepted as indicating an outcome evaluation. 
Although a significant number of programs did mention the measurement of child outcomes - 
and Figure 3.2 suggests that most MSRP programs do employ a child assessment instrument of 
some type - most of these programs did not report outcome results in their Narrative Summaries. 
This suggests that while programs may be assessing children, they perform poorly in terms of 
aggregating this information into the kind of results which facilitates decision making at the 



Figure 3.3 



Evaluation Report Characteristics: MDE Narrative Summary Form, 1998-99 





No descriptive detail or outcome evaluation 
Descriptive detail, no outcome evaluation 
Limited outcome evaluation 
Outcomes evaluated and descriptive detail 



Note: To be judged an outcome evaluation, 
required two of the following evaluative 
criteria: (1) concept of an outcome/effect; 
(2) category/unit/measuremenl clearly 
delineated; (3) comparison 



program level. This interpretation rests upon the assumption that if programs had aggregated 
results, they would have reported them when asked for such evidence in their Narrative 
Summaries. We know for certain that over 60% of MSRP grantees do not even mention that 
change at the child level is occurring and only 10% include the minimal information to make any 
kind of judgement that the program might be producing its intended results. 
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Figure 3.4 suggests similar conclusions in relation to longitudinal reporting. Most MSRP 
programs in the state do not appear to effectively measure the child-level impacts which their 
programs produce in subsequent years, and when they do, programs do not typically aggregate 
from the individual to larger groups. In this instance, the absence/presence of child-level 
outcomes in the Narrative Summary differentiates between simple reporting on transition (what 
kind of classroom and array of services the child moves into following MSRP) and reporting on 

Figure 3.4 



Longitudinal Reporting Characteristics: MDE Narrative Summary Form, 1998-99 
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B Transition and descriptive performance 
1 I Quantitative transition and performance 



performance (referring primarily to social and academic outcomes). The majority of Narrative 
Summary reports contain follow-up information pertaining only to transition to elementary 
school during the year following MSRP. These additional observations on the relative dearth of 
child outcome data in Narrative Summary reports strengthen our conclusion that MSRP 
programs are not focused on child outcomes and consequently, are not translating such 
information into program decisions. (A similar conclusion can be drawn regarding local 
programs’ lack of use of Program Quality Assessment data.) 

Child outcomes should be particularly important in this context, given the fact that it is 
precisely teacher perceptions of child behavior, disposition, and ability in the kindergarten year 
which determine how ready the child is for school. One key issue that may explain the relative 
lack of longitudinal efforts is the weakness of linkage between MSRP classrooms and 
kindergarten and early elementary personnel. As we have traveled the state working with MSRP 
staff, we have consistently heard about the difficulties that preschool staff have in coordinating 
with public school staff- even though MSRP classrooms are often located in the same public 
school buildings. 

Evaluation Capacity and LEA Organization 

As the discussion of Narrative Summary data suggests, another significant consideration 
in the attempt to raise the capacity of local education providers to evaluate program processes 
and outcomes is organizational. Program evaluations cannot be used productively when the 
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organization being evaluated is not coordinated around the effort. Local evaluations are often not 
methodologically suitable for high-stakes decision making (i.e., deciding future MSRP funding). 
This role is better given to the outside, statewide evaluation If, in addition, local evaluations do 
not serve a formative purpose at the agency level, then their value in their current form is truly 
questionable. 

Figure 3.5 shows that over 60% of those who sign off as the person completing the 
Narrative Summary form are neither lead teachers nor the education specialist associated with the 
local MSRP program. This fact may represent a significant organizational disconnect in the 
programs when the average number of lead teachers per grantee is well under 3. In our 
discussions with staff across the state on this matter, it is clear that evaluation responsibilities are 
not located in a consistent organizational role, and classroom staff are often unaware of the 
Narrative Summary form itself - the primary evaluative document in most programs. 



Figure 3.5 

Who Completes the MDE Narrative Summary Form, 1998-99 
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MSRP Program Evaluation - Part 1 Appendix Tables 

1. Cohort 1 Kg.: COR Total Scores by Group and Site Controlled for Key Covariates 

2. Cohort 1 Kg.: Analysis of Covariance for COR Total Scores and Subscale Scores 

3. Cohort 1 Kg.: SRRS Total Scores by Group and Site Controlled for Key Covariates 

4. Cohort 1 Gr. 1: SRRS Total Scores by Group and Site Controlled for Key Covariates 

5. Cohort 1 Gr.2: SRRS Total Scores by Group and Site Controlled for Key Covariates 

6. Cohort 1 Kg. / Gr. 1 / Gr.2: SRRS Item Scores for Program Effects Controlled for Key 
Covariates 

7. Cohort 1 Kg.: COR Subscale Scores by Group and Site Controlled for Key Covariates 

8. Statewide Risk Factor Data 1996: Distribution of Total Number of Risk Factors by Study Site 

9. Cohort 1 PreKg.: Total Scores for Program Quality Assessment (PQA) by Study Site 
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Appendix Table 1. Cohort 1 Kindergarten (Kg.): COR Total Scores by Group and Site 
Controlled for Key Covariates 



Site/Group 




MSRP 




Comparison 




Total 




rt 


Mean 


SD 


ft 


Mean 


SD 


n 


Mean 


SD 


Site A 


44 


3.39 


.46 


41 


3.56 


.39 


85 


3.47 


.43 


Site B 


43 


3.51 


.34 


40 


3.27 


.37 


83 


3.39 


.37 


SiteC 


58 


3.22 


.51 


36 


3.20 


.43 


94 


3.21 


.48 


Site D 


41 


3.73 


.61 


24 


3.44 


.52 


65 


3.62 


.59 


Site E 


24 


3.74 


.51 


35 


3.11 


.46 


59 


3.37 


.57 


All Sites 


210 


3.47 


.53 


177 


3.31 


.45 


387 


3.40 


.50 



Note. n=sample size; Substandard deviation. Key covariates = mother’s education, father at 
home, household size, gender and age. Analysis of covariance found statistically significant 
effects fox program (p<.001), site (pc.OOl) and program by site (pc.001) on COR Total scores. 
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Appendix Table 2. Cohort 1 Kindergarten: Analysis of Covariance for COR Total 
Scores and Subscale Scores 





Evidence of Significant Effects on COR Total and Subscale Scores 


Covariate 


Initiati 

ve 


Social 

Relatio 

ns 


Creative 

Represent 

ation 


Music 

& 

Movem 

ent 


Langua 
ge & 
Literacy 


Logic 

& 

Math 


COR 

Total 


Program 


pc.OOl 


p<.001 


p<.01 


p<.01 


p<.001 


No 


p<.001 


Site 


/7C.01 


/7C.01 


No 


pc.Ol 


p<. 05 


p<.001 


p<.001 


Program by 
Site 


p<.05 


p<.01 


p=.011 


p<. 01 


/7<.001 


p<M 


p<.001 


Gender 


No 


p<.001 


p=. 062 


p=. 060 


No 


No 


p<. 05 


Program by 
Gender 


No 


No 


No 


No 


No 


No 


No 


Age 


p<.05 


No 


No 


No 


No 


p<.05 


No 


Mother’s 

education 


No 


No 


No 


p=.071 


No 


No 


No 


Father at home 


No 


p=.061 


No 


No 


p<.05 


No 


No 


Household 


No 


p=. 053 


No 


No 


p<.01 


No 


No 


size 
















Amount of 
variance 
explained (R 2 ) 


12.4% 


15.5% 


10.4% 


14.6% 


19.5% 


26.5% 


20.3% 



Note. Significant effects: p<05; Nearly significant trends: . 10>p>.05; No= No significant effects: 
p>.10 . The sample size (Ns) are between 377 and 387. 
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Appendix Table 3. Cohort 1 Kg.: SRRS Total Scores by Group and Site Controlled for 
Key Covariates 



Site/Group 




MSRP 






Comparison 






Total 




n 


Mean 


SD 


n 


Mean 


SD 


n 


Mean 


SD 


Site A 


31 


2.70 


.33 


38 


2.56 


.48 


69 


2.63 


.42 


Site B 


35 


2.75 


.34 


45 


2.54 


.54 


80 


2.63 


.47 


SiteC 


48 


2.38 


.54 


28 


2.49 


.41 


76 


2.42 


.50 


Site D 


40 


2.53 


.48 


25 


2.63 


.36 


65 


2.57 


.43 


Site E 


30 


2.69 


.38 


0 


— 


— 


30 


2.69 


.38 


Site F 


28 


2.77 


.29 


16 


2.54 


.39 


44 


2.68 


.34 


All Sites 


212 


2.61 


.44 


152 


2.55 


.46 


364 


2.59 


.45 



Note. n=sample size; SD=standard deviation. Key covariates = mother’s education, father at 
home, household size, household income, gender and age. Analysis of covariance found 
statistically significant effects for site (pc.OOl) and nearly significant trends for program by 
site (p=. 069) on kindergarten SRRS Total scores. 



Appendix Table 4. Cohort 1 Gr.l: SRRS Total Scores by Group and Site Controlled for 
Key Covariates 



Site/Group " 




MSRP 






Comparison 






Total 




n 


Mean 


SD 


n 


Mean 


SD 


n 


Mean 


SD 


Site A 


43 


2.28 


.55 


35 


2.48 


.49 


78 


2.37 


.53 


Site B 


40 


2.44 


.44 


32 


2.24 


.54 


72 


2.35 


.49 


Site C 


39 


2.44 


.43 


25 


2.47 


.37 


64 


2.45 


.40 


Site D 


35 


2.52 


.46 


23 


2.46 


.48 


58 


2.50 


.46 


Site E 


28 


2.58 


.43 


9 


2.60 


.48 


37 


2.59 


.44 


Site F 


25 


2.54 


.44 


14 


2.25 


.48 


39 


2.44 


.47 


All Sites 


210 


2.45 


.47 


138 


2.40 


.48 


348 


2.43 


.47 



Note. «=sample size; SD=standard deviation. Key covariates = mother’s education, father at 
home, household size, household income, gender and age. Analysis of covariance found no 
significant effects for program, site, or program by site on Gr.l SRRS total scores. 
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Appendix Table 5. Cohort 1 Gr.2: SRRS Total Scores by Group and Site Controlled for 
Key Covariates 



Site/Group 




MSRP 




Comparison 




Total 




n 


Mean 


SD 


n 


Mean 


SD 


n 


Mean 


SD 


Site A 


33 


3.03 


.65 


28 


3.04 


.64 


61 


3.04 


.64 


Site B 


28 


3.31 


.65 


18 


3.09 


.61 


46 


3.22 


.64 


Site C 


37 


2.96 


.62 


26 


3.06 


.62 


63 


3.00 


.62 


Site D 


27 


3.13 


.59 


23 


2.78 


.88 


50 


2.97 


.75 


Site E 


21 


3.11 


.72 


6 


3.48 


.32 


27 


3.19 


.67 


Site F 


10 


2.96 


.86 


8 


2.64 


.84 


18 


2.82 


.84 


All Sites 


156 


3.09 


.66 


109 


2.99 


.70 


265 


3.05 


.68 



Note. «=sample size; S£)=standard deviation. Key covariates = mother’s education, father at 
home, household size, household income, gender and age. Analysis of covariance found 
significant effects for site (p<. 05), but no significant effects for program or program by site on 
Gr.2 SRRS total scores. 



Appendix Table 6. Cohort 1 Kg. / Gr.l / Gr.2: SRRS Item Scores for Program Effects 
Controlled for Key Covariates 



SRRS Item 


Grou 




MSRP 




Comparison 


Statistical 


P 

/ 

Grade 


n 


Mean 


SD 


n 


Mean 


SD 


Significance 




Kg. 


212 


2.47 


.74 


152 


2.31 


.76 


p <.05 


Retains learning 


Gr.l 


210 


2.51 


.71 


137 


2.38 


.77 


p =.077 


Ready to learn 


Kg. 


211 


2.65 


.57 


152 


2.53 


.66 


p <.05 


Gr.l 


210 


2.57 


.64 


138 


2.49 


.71 


^5 

\\ 

IO 

■-J 


Good attendance 


Gr.l 


207 


2.70 


61 


135 


2.61 


.69 


p <.05 


Interest in school work 


Gr.2 


156 


3.15 


.73 


109 


3.04 


.89 


p <.05 


Physical ability 


Gr.2 


152 


3.31 


.63 


107 


3.12 


.68 


p <.05 


Social/emotional 


Gr.2 


155 


3.09 


.85 


109 


2.88 


.87 


p=. 113 



Note. 7j=sample size; SD=standard deviation. Key covariates = mother’s education, father at home, 
household size, household income, site, age and gender. 
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Appendix Table 7. Cohort 1 Kg.: COR Subscale Scores by Group and Site Controlled for Key Covariates 
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Note . Comp.= Comparison group; n=sample size; M= mean; SD=standard deviation. Key covariates = mother's education, father at home, household size, gender and 
age . Significant effects: p<.05; nearly significant trend: . 10>p>.05. 



Appendix Table 8. Statewide Risk Factor Data 1996: Distribution of Total Number of 
Risk Factors by Site 





Total Number of Risk Factors Per Child 


Total Number 
of Children 


Site 


1-2 


3 


4 


5 or more 


Site A 


8 (10%) 


16(21%) 


21 (27%) 


32 (42%) 


77 


Site B 


348 (24%) 


371 (25%) 


284 (19%) 


470 (32%) 


1473 


Site C 


4 (4%) 


12(11%) 


23 (22%) 


67 (63%) 


106 


Site D 


38 (12%) 


95 (31%) 


88 (29%) 


85 (28%) 


306 


Site E 


51 (24%) 


70 (32%) 


48 (22%) 


48 (22%) 


217 


Site F 


54 (37%) 


33 (23%) 


32 (22%) 


26(18%) 


145 


Total 


503 (22%) 


597 (26%) 


496 (21%) 


728 (3 1 %) 


2324 



Appendix Table 9. Cohort 1 PreKg.: Total Scores for Program Quality Assessment 
(PQA) by Study Site 



Site 


n 


Mean 


SD 


Minimum 


Maximum 


Site A 


5 


4.14 


.33 


3.70 


4.56 


Site B 


9 


4.55 


.36 


3.82 


4.90 


Site C 


11 


4.34 


.14 


4.15 


4.55 


Site D 


11 


4.74 


.18 


4.27 


4.89 


Site E 


7 


4.59 


.08 


4.48 


4.71 


Site F 


6 


3.60 


.28 


3.21 


3.95 


All Sites 


49 


4.39 


.42 


3.21 


4.90 


Note. n=sample size; SD=standard deviation. Analysis of variance found statistically 
significant effects for site (pc.OOl) on PQA total scores. 
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